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1. INTRODUCTION 

In contrast to traditional classroom instruction, online learning challenges the traditional teaching 
model by giving students access to learning resources, and a variety of learning strategies. It enables students 
to increase and expand their knowledge regardless of location or time [1]. To accomplish the purpose of 
online learning, learners must have a better understanding of themselves and precisely identify their learning 
needs and cognitive level. 

Educational psychologists developed a system based on learning objectives that uses the taxonomy's 
levels to categorize students’ level of comprehension and learning [2]. The three domains of Bloom’s 
taxonomy are cognitive, emotional, and psychomotor. Because it is so closely related to students’ 
understanding and knowledge in the classroom, our study has concentrated on the cognitive domain at this 
point. Quick assessment analysis of a learner’s cognitive level during the learning process enables them to 
grasp their cognitive level and promptly alter their learning methods and materials. Additionally, it can assist 
teachers in providing timely, individualized instructional recommendations/interventions, more accurate 
implementation of lesson plans and structures, and supervising the learners’ general cognitive level across 
their learning journey [3]. 
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Learners’ cognitive level analysis requires manual coding, which demands a high level of research 
aptitude and expertise from the analyst. It is also a very time-consuming task. Additionally, Due to the 
enormous amount of information that needs to be sifted through, many researchers view manual analysis of 
textual data as a tedious and labour intensive effort. The manual coding approach falls under the category of 
shallow learning, necessitating the manual selection of many data attributes, being labour- and time intensive, 
and being limited in its ability to generalize [4]. Remarkable progress has been attained with the widespread 
use of artificial intelligence (AI), specifically machine learning (ML) techniques for classification tasks. ML 
constitutes a novel and recent technique for data analysis, it has promising outcomes and substantial abilities 
[5]. Complex data representation can be automatically learned and added to the analysed model [6]. From 
this perspective, the purpose of this research aims to automatically assess the learner’s cognitive level in 
online assessments and explore factors that impact the learner’s cognitive level during the learning process 
using the ML approach. 

The rest of the paper is organized as follows: section 2 summarizes the Bloom of taxonomy and a 
literature review about cognitive level extraction in online learning. Section 3 presents our structure, steps 
and method based on data analysis and ML. We demonstrate the experiment, the analysis and the discussion 
of the results in section 4. Finally, section 5 highlights the conclusion and discusses future work. 


2. LITERATURE REVIEW 

Several works use cognitive-level in e-learning to enhance the learning process. For a better 
understanding of the concepts, the first part of this section highlights an overview of Bloom's taxonomy. On 
the other hand, the second part is dedicated to presenting some recent works that use Bloom's taxonomy to 
model the cognitive state of learners and predict their cognitive-level. 


2.1. Bloom of taxonomy 

Taxonomy of educational objectives is a framework for classifying statements of what we 
intend/expect learners to learn as a result of instruction. All six of the primary categories in the cognitive 
domain have well-crafted definitions in the original taxonomy: evaluation, synthesis, analysis, application, 
comprehension, and knowledge [7]. The general criteria for programming evaluation as well as the hierarchy 
of Bloom's cognitive levels are displayed in Figure 1. Within this hierarchy, the categories were ordered from 
concrete to abstract and from simple to complex, where each lower level serves as a foundation for higher 
levels, meaning that proficiency at one level is necessary to advance to the next, more complex/difficult level 
[8]. Furthermore, a student who has attained a higher level has passed the previous levels with success. In 
order to facilitate categorization, we are using a variation of the cognitive process's three stages for this study 
[9]: high cognitive (CH) level (synthesis and evaluation), medium cognitive (CM) level (application and 
analysis), and low cognitive (CL) level (knowledge and comprehension). 


2.2. Cognitive level 

Troussas et al. [10] suggested a novel paradigm for the students' cognitive states' automated 
reasoning. The revised Bloom taxonomy and the k-nearest neighbors algorithm are used to show the 
cognitive states, which correspond to increasing levels of cognitive complexity. Additionally, a progressive 
approach based on Bloom's taxonomy was utilized to prepare the assessment questions, and the proficiency 
level was determined by that method. In this direction, Ullah et al. [11] propose a novel approach to 
programming assessment, which shows that Bloom’s taxonomy is a beneficial tool for learning and assessing 
programming. It achieves a student mapping to the respective cognitive levels of Bloom’s taxonomy directly 
from the written programming code, with no prior mapping questions. With an accuracy of 70%, Jayakodi et 
al. [12] used the cosine and WordNet similarity with natural language toolkit (NLTK) algorithms to create a 
distinct set of criteria that determine the test question's weight and category based on Bloom's taxonomy. Na 
et al. [13], in order to investigate students' impressions of formative assessment through a survey, it is 
proposed to create formative assessment as an instructional technique in real-time online classrooms and to 
investigate the use of Bloom's taxonomy in the production of formative assessment. Based on cognitive 
theory and fuzzy logic, Chrysafiadi et al. [14] suggested a fuzzy rule-based reasoner that determines the 
required number and degree of difficulty of test items for each level of the revised Bloom taxonomy in order 
to generate a customized exam. The learners' knowledge level is described using fuzzy logic, and the 
assessment results demonstrate extremely excellent accuracy in the test item selection for each individual 
student. Also, using fuzzy restricted boltzmann machine and natural language processing, Sairaj and 
Balasundaram [15] outline a novel approach for automatically creating questions based on Bloom's cognitive 
levels. In order to implement assessments to measure students’ cognitive levels, Abduljabbar and Omar [16] 
presented a novel technique that uses a combined strategy based on a voting algorithm and ML classifiers to 
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automatically categorize test questions based on Bloom's taxonomy's cognitive levels. In the same direction, 
Huilan et al. [17], focuse on the cultivation of students’ critical thinking through questioning and introducing 
Bloom’s taxonomy and the structure of observed learning outcomes (SOLO) taxonomy. It discusses how to 
make students-centred and how to train their critical thinking in the training process and cultivate students’ 
critical thinking to accomplish it. Pallegama et al. [18] presented a really intriguing concept by talking about 
an automated system with the ability to ensure that the teaching content and learning activities are within the 
learning outcomes, as well as to validate the learning outcomes and their levels for each module and lesson 
based on Bloom's taxonomy. Also, Subiyantoro et al. [19] concentrate on applying the Bloom taxonomy to 
determine the cognitive categorization system. They divide cognitive levels into three divisions using 
learning vector quantization (LVQ); low cognitive (CL), medium cognitive (CM), and high cognitive (CH). 
From another perspective Sukajaya et al. [20] use Bloom's taxonomy based serious game (BoTySeGa) as a 
substitute evaluation tool to gauge learners’ proficiency in mathematics. Mohammed and Omar [21], provide 
an automated test question classification technique based on the extraction of two features: word2vec and 
TFPOS-IDF known as, term frequency (TF)-inverse document frequency (IDF) based on part-of-speech 
(POS). Also, in order to analyse failed student behaviour and identify their needs, Wu et al. [22] created three 
classification models using data from two sizable online forums, then used the models to categorize postings 
in a private Facebook group into topics connected to statistics and those that weren't. Heilporn et al. [23] 
provided a thorough and all-encompassing picture of the methods used by instructors to improve student 
engagement in blended learning, based on cognitive and emotional behavior. A suggested intelligent 
framework by Barbosa et al. [24] makes use of the random forest (RF) classifier and ML methods to deal 
with the problem of determining which e-learning materials are appropriate for students based on their degree 
of domain expertise. 

Those presented methods elaborate the use of Bloom’s taxonomy in many e-learning areas to 
classify cognitive levels. However, those methods are generally based on complex structures and can not be 
generalised into any e-learning platform. We consider evaluation and online assessments to be the best tools 
to measure learner’s knowledge and comprehension evolution during the learning process. Moreover, the 
proposed methods did not take into consideration variables related to the method of resolving online 
assessments. Also, some affective traits and epistemological emotions like confusion or concentration are 
very essential variables in the e-learning process since they affect directly the learner’s online evaluation 
results and thus learning outcome. 


Evaluation Develop, formulate, discuss, design new concepts 


Synthesis Support, defend, judge, critique decisions 
Analysis Organize, compare, examine connections 
Application | Demonstrate, solve, execute information of new situations 
Comprehension Describe, explain, discuss cconcepts and ideas 


Knowledge Define, memorize, state basic concepts 


Figure 1. Bloom’s cognitive level 


3. METHODOLOGY 

Our methodology is to create a ML model that takes into consideration many affective and cognitive 
features in the context of online assessment. Figure 2 shows an overview of our suggested method. Our 
approach is broken down into three stages: the definition and aggregation of features, the ML model, and the 
decision-making. In the next subsections, we will elaborate in detail each step individually. Starting with 
defining our variables/features that could determine the learner's cognitive level in the learning process, the 
ML approach that will next classify each learner based on those features into different classes to decide at last 
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the accurate cognitive level of each learner and come up with the suitable pedagogical proposition for each 
learner in the system. 


Data definition and aggregation Data analysis & machine learning model Decision Making 


Concept acquisition D) Time spent . Hint count 7 ; Rone a — <. 
assessment = IF ining \" J | Cognitive Level í 
4 a sa e Low 


Figure 2. Methodology steps 


3.1. The definition and aggregation of features 

Since our method is based on online learning assignments, it was hard to carefully choose and 
aggregate suitable features that may contribute to defining the learner’s cognitive level. The main challenge 
in this research problem is that the online assignment questions are not properly categorized and correct 
weights are not assigned for each assignment/category/topic in the final decision in scoring learners. 
Moreover, we can not correctly define learners from score or yield in those assignments. Therefore, we 
propose a combination of affective and cognitive traits that affect Bloom’s cognitive levels of each learner 
during the learning process on a defined learning concept. We define those variables as (a)-(f): 

a. Time spend: it is the time between the start of the assignment and the final sub mission of the answer. It is 
an important variable to define whether the learner passes a considerable time or a short among of time- 
solving the problem. Generally, learners with high cognitive levels/engagement are more likely to solve 
their problem assignments in less time than others. 

b. Hint count: the number of used hints on the assignment. Every learner can use a number of hints to 
facilitate solving the assignment problem. The fewer hints used the higher Bloom’s cognitive levels of the 
learner. 

c. Attempt count: the number of wrong attempts to solve the problem. Every learner can use a number of 
attempts to solve the given problem. Similar to the hints, the more attempts used, the lower Bloom’s 
cognitive levels of the learner. 

d. First action: the first action determines the intuition of the learners and his/her cognition level. The 
available actions are: an attempt or ask for a hint. 

e. Type: this characteristic identifies the kind of the issue set's head section. Typically, each problem set is 
one of the three that follow: 

- Linear: the student completes each task in the prescribed order. 

- Random: the student completes all tasks/problems, however, the order of the problems is varied for 
every learner (randomly compared to other learners). 

- Mastery: random order, in which before moving on, the learner must "master" the given issue by 
correctly answering a certain number of questions in a row. Based on this classification, we can 
determine whether the learner attempts to randomly solve the problem with no cognitive 
engagement/level or whether he/she is cognitively engaged to solve the problem in a predetermined or 
correct order. 

f. Confusion score: although it is evident that the emotional and cognitive systems operate separately, they 
nonetheless have an impact on one another. One affective condition that links emotion and cognition to 
epistemic emotion is confusion. It is seen as an absolute cognitive state and an emotional state [25], but 
not an emotion [26]. Bloom’s cognitive levels are directly related to the confusion score of the learner 
during solving the problem. The learner with a high confusion score (confused learner), is generally a 
learner with a low cognitive level. Therefore, the confusion score is an additional feature, yet very 
important to determine Bloom’s cognitive levels of learners. 

All those features are gathered, cleaned, and combined to construct the dataset input into the ML 
model. It is crucial to remember that data is collected and grouped by a specific learning concept. The aim of 
the study is to learn the cognitive level of each learner around a learning concept in order to be able to 
analyse the learner’s behaviour precisely and anticipate future decisions. 


3.2. Data analysis and machine learning model 


It is crucial to remember that data pre-processing, a straightforward yet crucial step that ensures data 
consistency for better analysis, it must be done before data analysis. In this step, data is filtered by removing 
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redundant and irrelevant features, which is known as noise removal. Data analysis is the process of gathering, 
constructing, and expressing data in order to gather useful information, identify patterns of learning, and 
forecast future behaviour. This is a crucial step to the ML model. Over the several learning periods of the 
courseware, building a precise predictive system requires processing and representing the input at each 
learning phase. For instance, the data are interpreted using statistical metrics. To reduce dimension, feature 
aggregations and correlations are employed, and ML techniques are utilized for prediction. 

ML is an essential part of AI. ML is defined as “the field of study that gives computers the ability to 
learn without being explicitly programmed” [27]. It is a system of learning through examples. ML intends for 
a machine to deal with new situations through observations, self-training, and experiences in order to carry 
out particular tasks. ML is increasingly widely used in many aspects of daily life, including e-learning [5], 
[28]. The three main types of ML are unsupervised learning, supervised learning, and reinforcement learning. 
Reinforcement learning makes a series of decisions over time to maximize performance, whereas 
unsupervised learning evaluates input X without supervision from outcome Y. Input data X is used in 
supervised learning to predict the label of output data Y. 

After the creation of data vectors illustrating the cognitive state of each learner through time. In 
order to categorize data based on attributes and features, we plan to conduct a supervised ML study to find 
Bloom’s cognitive levels classes of learners. More specifically, we want to organize items into “classes” with 
the highest possible affinity within each class. Based on the supervised learning approach, we model the 
input vector X as the features presented in section 3.1 to predict the output data Y the present three classes of 
learner Bloom’s cognitive levels: high level, medium level, and low level. 

We first conducted a guided study to investigate a collection of well-known classification models in 
order to choose the top-performing one since it is improbable that the top-performing classification model 
can be found beforehand. We intend to examine a variety of ML models and choose the best models from 
them in accordance with the precision and accuracy of the findings. 


3.3. Decision-making 

Following the extraction of each learner’s cognitive level. The given cognition categorization model 
can make the supervisory process easier. In reality, adding the ML model to the e-learning platform can 
significantly improve the quality of decisions and learning alternatives. 

The design of the proposed method addresses the problem of the cognitive level of learners and the 
ability to predict it earlier through online assignments. The idea of analysing and predicting a learner’s 
cognitive level can shed the light on the cognitive engagement of the learner during the learning process. 
Tutors or the instructional designers who set up a known cognitive level before the start of the learning 
process can supervise learners who can not successfully pass the required lower cognitive knowledge level 
and thus come up with the best solution/intervention to help learners overcome their learning difficulties. 

This strategy intends to simplify the entire learning process by effectively automating logical 
decision-making activities with no need for expensive training intervention. Recent studies in the field of 
cognitive analysis are attempting to make use of ML techniques’ capacity for automating judgments. This 
opens up several study areas that have already been studied in the literature, including learning objects 
recommendation [29], questions classification [30], information retrieval from multimedia data storage [31], 
and summative assessment classification [32]. 


4. EXPERIMENT 

In order to validate the proposed approach. In this section, we manage the experiment to give details 
about our approach and verify its practicality. This section is divided into three subsections ; the used data, 
the experiment settings of the solution and last, the results and discussion. It is important to note that the 
dataset (after processing) and the source code are publicly available in [33]. 


4.1. Data 

We adopt the dataset drawn from the ASSISTments data for the school year 2012 - 2013 with affect 
predictions [34], [35]. We reorganized the dataset by selecting, cleaning, and aggregating relevant features to 
match our methodology. We chose “addition and subtraction of fractions” as a learning concept to study the 
cognitive acquisition level of learners for this specific concept. The final collected dataset contains 151016 
entries of different learners in the systems. The dataset was labelled into three classes (1 for low cognitive 
acquisition level, 2 for medium cognitive acquisition level, and 3 for high cognitive acquisition level). The 
dataset is annotated by three professional coders (a senior Ph.D. student and two full professors). The coders 
explained the annotation approach and rules prior to beginning the coding process. Cohen's Kappa was used 
to calculate the inter-rater agreement for the corpus that the two coders annotated [36]. We obtained a 
significant inter-rater agreement (Cohens kappa = 0.83) [37]. The discrepancy between low and medium 
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cognitive levels was usually the cause of the dispute. Ultimately, a compromise was reached by the three 
programmers. Table 1 exhibits examples of entries from the final dataset. 


Table 1. Examples of entries from dataset 


Attempt Hint First ‘ Time Label 
Learner Type À Confusion 
count count action spent score 
id0 3 4 Mastery Hint 0.2704 45 Low (1) 
id14 1 0 Linear Attempt 0 18 Medium (2) 
idl 1 0 Linear Attempt 0 7 High (3) 


4.2. Experiment settings 

The experiment was conducted using Python3 programming language alongside Pandas and Numpy 
libraries. There were 15182 unique items left after pre-processing, cleaning, and duplication removal for ML 
algorithm analysis. The training and test sets are split randomly into 12145 and 3037 entries, respectively, in 
accordance with the 80:20 rule. To complete the categorization tasks, Scikit-learn was used. An open-source 
Python ML library [38]. Numerous ML methods are included in it (classification, clustering, and regression). 

We compared the following candidate ML models: logistic regression (LR) [39], support vector 
machines (SVM) [40], gradient boosting trees (GBT) [41], naive bayesian (NB) [42], RF [43], and neural 
network (NN) [44]. In order to assess the effectiveness of the potential ML models, we adopted the following 
five distinct metrics: the recall, accuracy, precision, and F1 score. It is important to state that all experiments 
were conducted in Google Colab, with Tesla P100-PCIE-16 GigaBytes GPU and 25 Giga-Bytes RAM 
support settings. 


4.3. Results and discussions 

The purpose of the implementation is to train the proposed model and evaluate its efficiency and 
accuracy. This section compares the effectiveness of several ML techniques. The top results across the 
studied datasets are listed in Table 2. We notice that, among all ML techniques, the NN algorithm achieves 
the best performance. NN achieves an accuracy of 82.2% with a relative improvement of 0.3% compared to 
GB and 2% compared to SVM and LR. NB shows low accuracy results compared to the other ML 
techniques. 82.2% as classification result is quite considerable results in classification-based issues. This 
demonstrates the effectiveness of our model in detecting learners’ cognitive levels and the high performance 
of NNs compared to other ML. 


Table 2. The prediction analysis's outcomes 


F1 score Accuracy Precision Recall 
SVM 0.7486 0.8086 0.6996 0.8086 
NB 0.6560 0.6509 0.7164 0.6509 
GB 0.7632 0.8192 0.7168 0.8192 
LR 0.7498 0.8093 0.7003 0.8093 
RF 0.7626 0.8205 0.7123 0.8205 
NN 0.7654 0.8221 0.7188 0.8221 


The results are very promising. We can now predict the learner’s cognitive level with an accuracy of 
82% of each learner in the system through an online learning assessment. This can help the tutor and 
instructional designer to build better learning content for each category of learner (adaptive learning). Also, 
learners with low cognitive levels could benefit from a recommendation system with materials that match 
their low cognitive level. On the other side, learners with a high cognitive level could also profit from a 
recommendation system that challenges their high cognitive level. We can also enhance learners’ learning 
acquisition by grouping them into homogeneous collaborative groups that match their cognitive level. The 
power of predicting learners’ cognitive level can be the key to minimizing the high drop-out rate on online 
learning platforms, since the cognitive level and background knowledge are one of the main dropout factors. 


5. CONCLUSION 

In this research, by coding online learning concept assessments into three classes based on Bloom’s 
cognitive levels of each learner, we want to create an automatic system based on ML for evaluating learners’ 
cognitive levels. We attained a classification accuracy of 82.2% using NN algorithms. Our approach is 
essentially based on learners’ interaction in assessment in online learning to analyse and assess their critical 
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thinking and cognitive behaviour. As a result, this result will assess tutors and instructional designers in 
building better learning content for each category of learner and supervise learners with low cognitive levels 
during their learning process. Future works will focus on integrating this model into an online 
recommendation system where we can evaluate and analyse not only the cognitive level factors of learn ers 
but also other learners’ social and compartmental behaviour during their learning process. 
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